Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache slow/computed keys using an LRU #65

Merged
merged 1 commit into from
May 30, 2024
Merged

Cache slow/computed keys using an LRU #65

merged 1 commit into from
May 30, 2024

Conversation

mikejholly
Copy link
Contributor

@mikejholly mikejholly commented May 27, 2024 •
edited
Loading

This PR adds caching around slow cache key calculation and moves result.Release's into goroutines. The former change leads to minor benefits while the latter significantly speeds up cached build performance (nearly 10x). Cached builds that have a lot of dependencies & do many small tasks (e.g., our Earthly-in-Earthly tests) were formerly somewhat slow.

-- earthly-next

Fresh builds
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   11.06s user 3.65s system 9% cpu 2:31.86 total
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   11.02s user 3.22s system 8% cpu 2:44.52 total

Cached builds
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   14.44s user 4.07s system 6% cpu 4:31.65 total
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   14.58s user 4.69s system 8% cpu 3:57.24 total

-- With change

Fresh builds
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   11.22s user 3.40s system 12% cpu 2:00.82 total
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   10.82s user 3.29s system 10% cpu 2:12.73 total

Cached builds
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   11.20s user 3.23s system 37% cpu 38.148 total
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   11.24s user 3.28s system 54% cpu 26.827 total
earthly-dev --logstream-upload=false -P ./tests+ga-no-qemu-group1   11.16s user 3.03s system 52% cpu 26.899 total

Core tests: earthly/earthly#4164

@mikejholly mikejholly mentioned this pull request May 27, 2024
Closed
@mikejholly mikejholly marked this pull request as ready for review May 28, 2024 22:00
@mikejholly mikejholly force-pushed the mh/next-store-slow-cache branch 2 times, most recently from 67a59a4 to c98b808 Compare May 29, 2024 17:44
@mikejholly mikejholly requested a review from a team May 29, 2024 22:45
alexcb
alexcb approved these changes May 29, 2024
View reviewed changes
Copy link
Contributor

@alexcb alexcb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good; don't forget to update the earthly-next file under the root of the earthly repo.

@mikejholly mikejholly merged commit 88ecf5d into earthly-next May 30, 2024
30 of 53 checks passed
@mikejholly mikejholly deleted the mh/next-store-slow-cache branch May 30, 2024 00:52
@mikejholly mikejholly mentioned this pull request May 30, 2024
Merged
mikejholly added a commit to earthly/earthly that referenced this pull request May 30, 2024
@mikejholly
Update earthly-next to 88ecf5d (#4179)
1e0e0ef 
Updated to include: earthly/buildkit#65
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexcb alexcb alexcb approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mikejholly @alexcb